Heterogeneity and Causality: Unit Heterogeneity and Design Sensitivity in Observational Studies

Before R. A. Fisher introduced randomization, the literature on empirical methods emphasized reducing heterogeneity of experimental units as key to inference about the e¤ects of treatments. To what extent is heterogeneity relevant to causal inference when ethical or practical constraints make random assignment infeasible? 1. Notation and Review 1.1. Review: Randomization Inference in a Paired Experiment. Observed covariate x and an unobserved covariate u. I pairs, i = 1; : : : ; I, of two subjects, j = 1; 2, one treated, one control, matched for x, so xi1 = xi2, but not matched for u, so typically ui1 6= ui2. Zij = 1 if j received the treatment in pair i, and Zij = 0 if j received the control, so Zi1 + Zi2 = 1: Subject (i; j) has two potential responses, (rTij ; rCij), rTij observed under treatment, Zij = 1, rCij observed under control, Zij = 0, so the e¤ect of the treatment is rTij rCij ; Neyman [16] and Rubin [25]. Write F for f(rTij ; rCij ;xij ; uij), i = 1; : : : ; I, j = 1; 2g and Z for the event fZi1 + Zi2 = 1; i = 1; : : : ; Ig; then F and Z are …xed by conditioning in Fisher’s [5] theory of randomization inference. Randomization within pairs ensures Pr (Zi1 = 1 j Z; F) = 1 2 , i = 1; : : : ; I, with independent assignments in distinct pairs. Observed response is Rij = Zij rTij + (1 Zij) rCij . If treatment e¤ect is constant, = rTij rCij , then Rij = rCij +Zij , and the treated-minus-control di¤erence is Di = (2Zi1 1) (Ri1 Ri2) = + i where i = (2Zi1 1) (rCi1 rCi2). Test H0 : = 0 by ranking jDi 0j from 1 to I; then Wilcoxon’s signed rank statistic, W 0 , is the sum of the ranks for which Di 0 > 0, where ties are assumed absent. If H0 : = 0 is true, randomization ensures that Di 0 = i is rCi1 rCi2 or rCi2 rCi1, each with probability 12 , independently in di¤erent pairs. Given Z; F , if H0 : = 0 is true, then the jDi 0j are …xed, the Di 0 are independent, Pr (Di 0 > 0) = 12 , and each Di is symmetric about 0, soW 0 is the sum of I independent random variables taking values i or 0 each with probability 12 , i = 1; : : : ; I. A con…dence interval for is obtained by inverting the test, and the Hodges-Lehmann [8] estimate b of is (essentially) the solution to Wb = I (I + 1) =4 = 12 (1 + 2 + : : :+ I). Null distribution of W 0 is the same for all (untied) F , but the nonnull distribution depends on F or a model that generates F . A common model has (rCi1 rCi2) = iid F ( ) were > 0 and F ( ) is continuous and symmetric about zero, so randomization ensures i= iid F ( ). June 2005. Supported by Grant SES-0345113 from NSF.